After _sync_final_step() replaces each local step with the global MAX, this boundary may only have been dumped by the longest-running rank. For example, if ranks finish at steps 49 and 50, the shorter rank adopts 50 and skips here, so its trailing tracked rows are never written. Please reject uneven exhaustion for all multi-rank dump cadences, or base this skip on each rank’s actual last dump while preserving collective participation; add a 49/50 regression test.

This failure rendezvous is limited to minute cadence. In default step mode, rank zero can observe an async uploader failure immediately while peers skip the throttled marker check, raise alone, and leave peers hanging in the next training collective. Please synchronize the failure bit for every distributed FeatureStore cadence before raising; boundary dump failures need the same treatment.

Minute cadence executes this CUDA all_reduce plus host-synchronizing .item() calls every training step, while the enabled pipeline also all-reduces batch availability. That adds at least two blocking collectives per batch for dumps minutes apart. Please coalesce timer/error state with the existing availability rendezvous or otherwise remove redundant hot-path synchronization.

read_table() + to_pylist() materializes every embedding element as Python objects, then copies rows into NumPy arrays and another sorted list before uploading. One large delta can therefore exhaust rank-zero training memory despite max_pending_steps. Please stream Parquet batches and use bounded/spilled deduplication so memory is capped independently of one dump’s size.

The worker calls _add_discovered_steps_locked() on every poll. Because committed dumps/manifests are retained, each poll rescans all historical steps, rereads success/manifest JSON and Parquet schemas, and does so while holding _condition, so submit()/close() latency grows with job history. Please reconcile once at startup and incrementally track later submissions/new entries.

These canonical paths are sampled without validating their hashes/generation against the selected success marker. A restored run can replace the same step’s canonical files before the new generation commits, making this tool compare uncommitted local rows with the previous remote commit. Please verify the selected files against success["shards"], or fail explicitly when the committed bytes are no longer available.

Please source this credential-bearing SDK from an organization-controlled registry/release and pin its SHA-256 hash. A mutable wheel URL in a personal bucket makes dependency compromise equivalent to arbitrary code execution with the training job’s FeatureStore credentials.

This only rejects URI userinfo; an arbitrary plaintext or attacker-controlled endpoint is still passed to the SDK together with cloud and FeatureDB credentials. Please require HTTPS and trusted FeatureStore endpoints (or an explicit trusted custom-endpoint opt-in), and reject unexpected path/query/fragment components.

This says the version must be provisioned before upload, but the uploader contract is AUTO_CREATE_ON_FIRST_DELTA_MERGE and the implementation deliberately performs no version precheck. Please document that the explicit non-default version is created by the first MERGE.